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(57) The method and system o( the invention main- 
tains the indication of the togical relatkmship ot f reef om> 
digital ink annolatkxis to objects in an eleclronc display 
even as the view changes. The inventcn maps the tog- 
teal relationship ol the freelonn digital ink annotations 
to objecis in the view based upon the layout relation- 



ships of the freefomn digital ink annotattons to the ob- 
jects in the view. If the view changes, the tnventon man- 
tains the layout relatkmsh^p of the freeform digital hk 
annolattons to the objects in the view based upon the 
mapping of the freeJorm digital ink annotations to the 
objects. 
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Description 

[0001] This applicalton b deeded to an electronic system for displayong changing views. In particular, the invention 
is directed to permitting f reeform distal (nk annotation on electronic media and to maintaaiing the lo^l relationship 

s betwem f reeform digital annotations and objects h changing views of the media 

[0002] Freefonn mk annotation of paper documents is a wida«pread pradice because fi aids many aspects of doc- 
ument work. Excluding the processes of thinkng, categorizing and reviewing. In short, freefonn ink anrxstatnn has 
proven to be a powerful comprehenston tooL Paper and paper-fike documents (such as PDF. postscript, image files, 
etc.) have a static layout and content. The static nature of these documents makes annotation simple. 

10 [0003] However, the kinds of documents that people use are fftcreasingly created and presented cfigitaJly on displays 
using computers. Such documents are often dynanuc and, therefae, the accurate representatk)n of the relationship 
of the anr)otatx)ns to objects in the display has been impossible because of the changing display. For example, a 
display of a word processor document changes as the document is undergoing revisksns. In a simitar mannei; infor- 
malkxi visualization systems such as the hyperbolic browser disctosed in A Focus ^ Context Technique Based on 

f 5 Hyperbolic Geometry for Visualiztng Large Hierarchies, by tamping et al.. in Proceedings cA ACM SIGCHI Conference 
on Human Factors and Computing Systems. May 1 995, change the display of the document as the user navigates the 
infomnation space. 

[0004] A freeform digital ink whSeboard known as Ttvoli is disctosed in Pert-Based Interaction Technkiues For Or- 
gan iztno Material on an Electronic Whrteboard. by Moran. et al.. in UlST '97. ACM press. Tivoli alk)ws users to identify 
a connectkm betwe^ two areas in a display, where that connection is maintained as those areas are moved and the 
display changes. Circle marks whk:h surround other ink identSy areas, and freefonn digital ink marks which connect 
tm area boundaries together form a connection between those two areas. Tttis connectkxi persists as the areas move 
and Tfvolr attempts to maintain the shape of the connecting mark. 

[0005] To assist in navigation of documents conventional systems use a set type of bookntark Examples of such 
25 navtgalkxi systems are disctosed in Virtual Annotatkyi Syslerri. by Harmon, et aL, in Proceedings of the IEEE 1996 
Virtual Bealrty Annual International Symposium, pp. 239-245 (t996) and in Wofkflets:3D Thumbnails for WSavfindIng 
in Virtual Environments, by EMns. et al.. in video proceedings of UlST "97. pp. 21-30 (1997). These systems make it 
ackward for a user to freely Irtterad with the navigation system because the user is required to specify an annotation 
command before commenting. 

00 [OOOq Current systems attempt to manage freefonn d^itat ink annotation on changing views or displays by at least 
one ol three methods (the temrs "view* and 'display* are used interchangeably throughout this applbatkin). The first 
method, known as the state approach, treats ink annotatk)ns as associated with a scngle view When the presentatbn 
or the content of the document ch^ges. the annotations either dsappear or remain tn their original positk)ns. A system 
known as 'MATE* which is disctosed in A Markmg Based Interface for Co»atx)rative Writing, by Haidock et at., in 

^ Proceedings of UlST *93 (1993). addresses changing content by maintaining a static artnotaled version, a changed 
verskin without annotatkxi and togical links between these two versions. 

[0007] A second method, known as the snapshot approach, treats the dynamic view as a senes of state snapshots 
which can be indivklually annotated. Examples of this approach include the W^g Freestyle disctosed in Rapid. Inte- 
grated Design of a Multimeda Communication System. by 'Francik. in Human-Computer Inlerface Design, Morgan 

40 Kaufman Pubfishers. Inc.. (1996). 

[0008] The third method, known as the tenrporal assoctatton approach, separates the ink annotations from the view 
and links the two temporally, so that marks made in a specffic time are associated with the view at that time. Examples 
of this approach are used in Marquee whtoh is disctosed in Marquee: A Tool for Real-Time Vtdeo Loggino, by Weber 
el al. , in Proceedings of CH\ *94. ACM Press. (April 1 994) and Dynomite which is disctosed n Dynomite: A Dynamically 

^ Organized Ink and Autfio Nrtebook. by Schilit et aL, In Proceedings d CHI ACM Press. (1 997), 

[0009] In conventtonal interfaces that accept freeform digital ink annotatton. the indtoatlon of the togtoal relationships 
between the annotattons and the associated objects are not mainlamed if the view changes. For exsnple. it a user 
annotates the text <^ a document and; subsequently, the annotated text reftows across lines, the annotatton does not 
foltow the reflowed portion of the text. Therefore, while there may be an underlying togicai relattonship to the annotated 

so text, the layout of the new view does not indicate this togtoal relationship between the text and the annotation. 

[0010] The systems described as related art above also do rvA maintain any indcation of togical relattonships be- 
tween the annotattons and the associated objects 9 the view changes. Tivoli does not maintah the freeform digital nk 
annotattons as the areas move around and the display changes. Rather. Tivoli maintains only one specific type ol 
. annt^ton: a stngto stroke from the boundary of one area to another. This restriction precludes the use of freeform 
(figital ink anm^ions in changing cfisptays to tabel moving objects or to akJ navigation. 
[001 1] While MATE maintains the togtoal retattonshfts between f reefonm dgital ink annotattons and the corresponding 
underlying text, it does not maintain the indicattoris these togical relationsh^ as the view changes. 
[0012] Freestyle does not altow the freeform (figftal ink annotattons to carry over to new views of die same document 
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[001 3] Dynomtta does not pemut annotations dtrectty on tho view and the annotations are not rendered on the new 
views In accordance with changes to the views. 

[001 4) The method and system d the cnvention maintain the layout relationships t)etween freeform digital ink anno- 
tations and objects tn changing views. As a user makes a f reetorm digital ink annotation on a view and as the view 
5 changes, the method and system of the invention adapt the marks in accordance w^ the changes. Thus, the k)gca) 
relationship between the freelorm digital ink annotatkans and the objects made in one view can be understood tn future 
views based upon the layout retatkmships. 

[001 5] The method and system also maintain the indicatnn of the logical relatksnsh^ between free! onm digital ink 
annotatbns and objects in dynamic media such as vkjeo and interactiva virtual workte. Pot example, if a user, who is 
10 exptorng an information space with a hyperbdic browser, makes a freeform digital ink annotatksn on a node dt the 
tree, when the user changes the display to brtng another portion of the tree into focus, the freeform digital ink annotation 
rotates and scales along with the tree to continue to indicate the togcal relationship of the freeform dgital ink annotatbn 
to the node. 

[001 6] The methods artd systems d this trwentton maintain the indication of the togical retattonships between f reef omi 
IS digital ink annotations and objects in a view as the view of the anrK)lated objects changes. This invenlbn brings the 
experience of marking on paper to dynarmc media, changing word^rocessor documents, biformatkm visualizations, 
virtual worlds, digital video and word processors. 

[0017] The system and method of the present invention maintain the hdicatton of the toglcal relationship, even if the 
•ot>jects that are togk^Ily related to the freeform digSal ink annotations are deleted, inserted, copied or moved or oth- 

20 envise altered The method and system also maintah the indbalon of the iogpcaU relatcnship if the layout presentatkv) 
of the objects in the display change because the objects reftowt scale, resize, rotate, stretch, distort or nrK>rph or if any 
other aspect of the layout presentation changes, regardless of Aether these changes occur discretely or continuously 
[0018] Additcnalty. another exemplary embodiment d the systems and methods d the invention provides naviga- 
tkinal markers that make freeform digital ink annotatnn a powerful tod lor navigating dynamk: documents. A user is 

^ penmitted to select a freeform digital ink anndatkn in an arbitrary vie^ and the display is retumed to the state it was 
In when the selected freeform digital ink annotatkxi was made. 

[001 9| The preferred embodiments d Ihs invention will be described in deta3. with reference to the folbwing figures, 
wherein: 

30 Fig. 1 is a bkx:k diagram d an embodiment d the system for displaying changing views d this tnventk)n; 

Fig. 2A shows a display d an annotated hypertx)lic browser in accordance with an embodiment d the present 
invention; 

Fig. 2B shows the annotated hyperbo&c browser d Fig. 2A with the view changed and freeform digital ink anno- 

tatkxis adjusted in accordance with the change in the view tn accordance with an embodiment d the inventkm; 
3S Fig. 3A shows a display of an annotated document; 

Fig. 3B shows the annotated document d Fig. 3A after modiftcation d the document and shows the acQustnoents 

to the freeform digital ink annotatbns in accordance with an embodiment of the present tnventnn; 

Figs. 4A-4F show a flowchart outlining a contrd routine for mapping f redorm digital ink annotatk)ns to objects in 

a display in accordance with an embodiment d the prevent invention; 
40 Figs. 5A-5B show a ftemchafl outfining a contrd roulffie for rendering f reefonn digital ink arviotatkxis on a changed 

view in accordance with an embodiment d the present inventkm; and 

Fig 6 shows a few exar^ptes d cfifferent types d freeform digital ink annotations. These and dher features and 
advantages d this inventkm are descrbed in or are apparent from the fdtowing detaileddescnpton d the prefened 
embocfiments. 

45 

[0020] Rg. 1 shows one exemplar embodiment d the electrcnte system 1 0 d this hventksa The electronic system 
10 includes a processor12communcating with a memory 14 that stores the programs, documents and other data for 
practicing the tnventkx). The processor 12 also communicates with a display 16, a k^board 18. a mouse 20 and a 
pen 22 via input/output device 24. The keyboard 18. mouse 20. pen 22 and any other interface devices (nd shown) 

^ are operated by a user to contrd the operation d the electronic system 10. Contrd d the electronic system 10 is nd 
Imrtited to the interlace devces shown and does not require any particular interface device. The display 16 may be on 
a stand-alone pen contputer or a tablet connected by a tether to a conventional computer or any other type d display 
[0021] As shown in Rg. 1 . the system 1 0 is preferably implemented using a programmed general purpose computet 
However, the system 10 can also be implemented using a special purpose confer, a programmed rmcroprocessor 

^ or microcontroQer and any necessary peripheral tntegrated circuit elements, an ASIC or other integrated circuit, a 
hardwired electronic or k>gk: circuit such as a discrete etemmit drcuft, a programmable logk; device such as a Pti), 
PtA FP6A or PAl. or the Kke. In general, any device on which a finite state machine capable d implementing the 
flow charts shown in Ftgs. 4A-5B can be used to implement the system 10. 
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[0022] Addftionaily, as shown h Fig. 1, the memofy 14 is preferably implemented using static or dynamic RAM. 
However, the memory 1 4 can also t>e trnplemented using a floppy disk and disk drive, a wrilabfe optica) disk and disk 
drive, a hard drive, a flash memory or the nke. Addttionalty. it should be appreciated that the memory 14 can be either 
distinct portions of a single memory or physically distinct memones. 
5 [0023] Funher, it shou U be appreciated that the Onks 26. 28. 30. 32. 34 and 36 con n ecting the memory 1 4, the display 
16. the keyboard IB. the mouse 20, the pen 22 and the input/output device 24 to the processor 12 can be wired or 
wireless links to networks (not shown). These networks can be toca\ area networks, wide area networks, intranets, the 
Internet or any other distr^ed processing and storage networks. 

[0024] Fig. 2A shows a display of a view of a hyperbolk: browser 40. The display 40 includes a f reelonm digital ink 
annolatcn 42. Fig. 2B shows an allemative view of the document shown in Rg. 2A. The freefomi digital ink annolatfon 
42 has been scaled and moved so that the layoul presentatron the freeform dgital ink annotatkan in the view corre- 
sponds with the change of the view of the objects in the document 40 to maintain the indication of the k)gical rebtionshfp 
between the freeform digital ink annotatkyi 42 and the objects. 

[0025] Fig. 3A shows a first view 50 of a document 52 having freeform digital ink annttotkins 54. 56 and 58. Fig. 38 

15 shows a second view 60 of the document 52 after the document has been revised Fig. 3B shows that the freeform 
digital ink annotatkins 54. 56 and 58 are ac^usted in accordance with the k)gcal relationships to the objects in the 
document 52 in accordance with an embodiment ol the inventnn. The content of the document 52 shown in Fig. 3A 
has been edited to insert the text "is so much fun to do' at 62 and to delete approximatety two fuB fines of text at 64. 
tn response to these edits, the text that has been annotated at 54 now ftows across two Ines and the 'crossouT 

20 treetomi dgital ink annotation 54 fkTws atong with the underlying text objects to continue to tndtoate the bgical rela- 
tkxtship between *cro6S-out' freeform digital ink annotation and the underlying text objects. The invention also recog- 
nizes that the group of ink strokes at 58 is a margin note and the system slightly adjusts the vertual positnn of the 
margin note 58 in the margin to continue to indcate the b^l retatkxish^ between the margin note 58 and the refbwed 
tine. Adcfitionalty, the freeform digital ink annotation 56 is adjusted to adapt to the one tine being removed between the 

2S anchor points at 66 and 68 to continue to irK&ate its logical relatxxiship to the text. 

[0026] An embodiment of the method and system of the present ffrventkxi involves at least a three-step process. 
The first step is adding the freeform digital annotatkxt on objects in a first view The second step involves mapping 
segments of the freefomi digital ink annotatkxis to corresponding ot^ects in the view The third step involves mo\dng. 
scaling or otherarise transforming the freef oim digital ink annotatkvi to maintain the indnatlon ol the logical relationship 

30 between the freefonn digital ink annotation and the corresporxling objects as the display changes. 

[0027] Optkx^ly* ^ embodiment of the inventkxi the freeform digital ink annotations can be used as navigational 
markers. For example, the user can select an annotation in a view and return to the annotated view that existed when 
the anrKJtatlon was created. 

[0Q2B] The freeform di^ ink annotatkx) of objects in changing views of an embocfiment of the invention relies on 
3S an object model for the view. In the ^cample of Figs. 2A ard 2B. the objects are the nodes in the hyperbole browsei: 
In the example of Rgs. 3A and 3B. the objects are the text characters. In a third example (not shown), the objects are 
the output from an object tracking system running on a vk^eo data stream. 

[0029] An embodiment of the method and system ol the invention maps the freeform dgital ink annotatioris to objects 
in a view fay placing anchors in the underlying document For example, a sffnple ctrdtng, highK^iting or a crossing out 
^ mark is anchored to the con-esponding span of character positkins that are covered by the freefomi digital ink annotation 
in the view. In the example shown m Figs. 3A and 36. a comment 56 in the margin is anchored to a positkm in the 
document in the margin near all of the corresponding words or characters. 

[0030] More complex freefonn digital ink annotatkxis may have several anchors. For example, as shown in Figs. 3A 
and 38. oonskler a freefonn digital ink annotatkxi 56 that includes a circle 68 connected to an anow 66. The circle 68 

45 is anchored to the span of words or charaders that is selected by the circle and the tip oft 
its end point while the line connectcng the circle 68 and the tip of the arrow 66 is nc4 anchored. 
[0031] An even more complex freefomi d*^ ink annc^tion may irwolve (ogical retatnnsh^ to moving objects in 
a virtual wodd. An exarnplaol such an annotation involves the freeform digital inkannotatkanofadoortoaroom^a 
virtual work! The layout of the door changes in a view of the virtual worki as the door is opened or cbsed. The method 

so and system of an embodiment of the tnventkxi maintavi the indk:at)on of the k)gk:al relationship between the freeform 
digital ink annotatbn and the door as the view of the door changes. Thus, the user sees the freefonn (figital ink anno- 
tation rotating with the door. 

[0032] For the purposes of the folkiwtng dtscussnn. the following definitkxts are providadt A positkin is a two<fimen- 
skxial coordinate bi a view. A stroke is a list of positions in a view. An offset is a twond'ff^^ 
ss in aview. A regno is a portxxi of atwodimenskanal view timt is represented, for example, by a bounding box or by a 
list of positnns. A character is a character in a text document Character positkxts between characters in the ftow of 
text are maintained as the text changes. A margin anchor is a character positkxi and a boolean flag for left or right 
margins. A simple anchor is a character, a character positnn or a margin anchor. A composite anchor is a pair of simple 
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anchors wfth offsets. An anchor is either a simple anchor or a composite anchor. An anchored stroke is an anchor plus 
a list of offsets. 

[0033] An emtxxJiment of the method and system of the invention moves the anchor points if the underlying anchored 
objects move. Unanchored segments of the freefomn digital ink will etretc^ to fit between the anchored points. Soma- 

5 times adjacent anchor points move apart because the underlying objects move apart For example, if a text phrase 
reflows from a single line across two lines (as shown in Figs. 3A and 36). In such cases, the conriection t)etW8en the 
formerly adjacent anchored points can be left out to reduce dutter and any unanchored ink between those points can 
be removed Thus, as shown in Fig. 36. the reftowed freefonm digital ink annotation is spfit and does not show ink 
extending between the hwo segments, which are on two separate text line& If anchor ejects for a stroke are deleted. 

'0 then the freeform digital ink annotattons can be remapped to the nearest remaining objects in the document. If aD 
anchor objects for the f reefonn d^itat ink annotatkxi are deleted then the annotation can bo deleted 
[0034] The type of the artchors for the freeform digital ink annotatkx) depends upon the shape of the annotation and 
the kind and number of the correspond'ng objects. One method, detailed below, for confuting ink anchors is to select 
the nearest object for each segment tn the ink. Altemativelyi the type of stroke, as determined from Its shape, can be 

75 used to determine the anchor objects. In another case, a freef<mn digital ink annotation with multiple anchors may 
cause unwanted distortkxis when the ^ew changes. To avoid this problera an embodiment of the present invention 
groups segments of the freeform <£gital ink annotation together to reduce the number of anchors. 
[0035] As bneffy described earlier, an embodiment of the method and system of the present ffwantion uses at lea^ 
three processes. The first process receives freeform di^ ink annotations on the vtewi the second process computes 

20 a logcal mapping between objects in the view and the freeform digital ink annotatkns. and the third process renders 
the freefomi digital ink annotations on a view that has changed. An optional fourth process selects freeform d^l ink 
annotations to navigate to a prevkxis state. 

[0036] The first process includes sensing data points from a pointing devne, such as a mouse or a pen, rendering 
the ink strokes by connecting the pomts on the v»w, and grouping the ink strokes together to torn a freeform digital 
25 ink annotation. These ink strokes may be grouped, for example, by a simple time threshoU. The second process 
receives the (reefonn digital ink annotatkxis as Input and maps between the objects and the freeform digital ink anno- 
tatkxis. The third process updates the view when the document changes. This process uses the mapping to render 
the treef onn digital hk annotatkxi on the objects as the view changes. 

[0O37] The procedures and the embodiments of the tnventkxi descrbedherein assunw that the objects do not change 
30 while the user is inking. However, the method and system of the inventkm is not limrted to appficalkxts in which the 
objects being annotated do not change. As an example, en the case of vtdso^ the video may be hatted while the user 
marks. Alternatively, the video can continue white the user marks and the state of the objects at some point in {me 
can be used to interpret the objects and update the mapping between the objects and the freeform digital ink annota- 
tkxis. 

3S [0030] The fourth process takes a seledksn on a vien determines if it conesponds to a freefomi cfigital ink annotation, 
and. if it does, replaces the current view with the view which existed when the freefomi digital ink annotatnn was made. 
This process requires storing an adcfitkxial mappmg from the freefomi digital nk annotations to the state of the changing 
view. 

[0039] The present invention recognizes that objects may be built from sub-objects wNch may be split from each 
^ other. One exampte is a text phrase that is buift out of words. If a phrase has a freeform digital ink annotation that 
circles the phrase and if that phrase refbws across two Ones, the inventkxi recognizes this split and may either reflow 
the freeform digital ink annotation by fitting it up into several segments, each of whk:h foOows the corresponding 
underlying words as they move, or the sy^em may choose to draw lines between matching segments of the circle. 
Alternatively, the freeform digital ink wnotatkxi couM be repTtcated at a smafier scale on each ondivkJual word 
45 [0040] A user can also copy a phrase to another paragraph, which couU be considered a form of splitting or as an 
entirely new and unrelated object, wthout the circle annotatkxt 

[0041] Conversely, ot^jects can merge into larger entries. For example, if an unannotated word between two anno- 
tated words is deleted then the two annotated words "merge* tofomi an annotated phrase. The two annotatkxis join 
to form a single annotation of the merged phrase. 
so [0042] Ot>jects may also disappear entirely. For example, words-tn a text editor can be deleted In this case, the 
freeform digital ink armotatkxts associated with the object can disappear also, or they can remain at the kicatcn be- 
tween the surrounding objects. 

[0043] Objects can also change. For example, a user in a texl editor can select an annotated word and replace it 
with a new word The new vtotd couU be conskfered a a new version of the orig^l object The annotatkm on the 
^ original word may then be associated wfth the new object In this case, the annotatkxi may alter its shape to conform 
tothenewword. for example, by stretching or shrinking. Alternatively, the annotatkxi could be renwved from the display. 
[0044] An embodiment of the method and system of this invention may group ink strokes on text to form a single 
freeform digital mk annotatkxt. 6y way of non-limiting examples, the annotatkxis can be categorized as circles, under- 
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lines/cfoss-outs, insertions, margin notes, and connectors. Annotations that are circles are identified by ink stroke 
circles that contain text and these ink stroke circles are mapped to the text objects inside the ink stroke circle. Undedine/ 
cross-out annotatkxis are identified t>y etongated marte which are either directly on the text or underneath the text 
These annotatnns are mapped to the text ob^cts that they are on or underneath. An annotation may be identified as 

s an tnsertkm 3 the annotations dips into a line of texL An insertion annotatkm maps to the position where the insertion 
mark appears in the text, for example, between two characters. A margin note is an annotation which is kxated in a 
margn. The margin note anr^tbns are mapped to the text that is adjacent to the vertical andfor horizontal pos^ron 
of the annotations. Connector annotations are identified as connectors by arrows that may appear at the end of the 
annotalkm or by lassos Q,q., a circle plus a One and a single stroke). A connector annotation is split into a ctrda and a 

10 connector portion. Connector annotations are mapped to two objects inctependentty The first object is the ot^ect that 
occurs at the start of an annotation and the second object occurs near the end of the annotatton. These objects may 
be positions in the text, or they may be other annotations. As an example, one end of the fine pottm of a lasso is 
mapped to the circle annotation whch. in turn, maps to the text. Another example is a line from the text to a margin 
annotatkin. One end of the line is mapped to the text whOe the other end is mapped to the margin annotatnn. 

>5 [0045] Rg. 6 shows several exemplary types of f reef orm digital ink annotations. Fig. 6 shows a display of a document 
400 including circle (or enclosure) amK>tations 402 and 404. an underline annotation 406. crossout annotations 408 
and 410. insertion annotatkyis 412 and 414. a margin annotation 416, a tine to nnargin annotation 416 (with associated 
margin note 420) and an arrow connector 422. It shouM be understood that the types of freelorm (figital hk annotations 
shown in conjunction with the descriptions of various exemplary embodiments herein are intended to be illustrative 

20 and not finuting to any particular embodirnent/type or grarnmar of annotation. 

[0046] The method arxi system of this inventbn is based upon the concept that anchors are determined for freelorm 
digital ink annotations. Anchors are offsets from objects. Freeform digital ink annotations are mapped to the object 
structure of the view underlying the f reeform digital ink annotatnns. While it is to be understood that the mapping may 
be to hierarchical object structures such as characters, phrases and paragraphs, lor the sake of s^nplicityi the (ottowing 

2S descr^tnn (Ascribes a mapping of a single level: between freeform digital ink annotatkxts and characters or positcns 
between characters. A text editor maintains these characters and positkms as the text changes. For characters that 
have been deleted, the text editor marks the characters and their con^esporxting character positions as deleted, but 
the text editor stiD maintains a correspondence between the character posilkxis and some Ipcatkin between remaining 
characters. 

00 [0047] An erribodimern of the rriethod and system of the invention nriay rely i^xan a text edit^ 

and presentation of characters. The embocfiment can determine if a cftaracter positkyi. a character or a range of char- 
acters has been deleted; can acquire the position h the view that correspor^ds to a character posSion; acquire the 
region in the view that corresponds to a character; can obtain left and right margin positions that correspond to a 
character position; can acquire a range of character positions that correspond to a region in a view; and can split a 

3S freeform digital ^k annotatkxi into a set of substrokes or segments, such that each substroke corresponds to a single 
character or to a positkxi between characters. 

[OO40| One embocfiment of the method and system of the invention may also use an annotation database that stores 
anchored strokes. The annotation database may be accessed to add an anchored stroke to the database, to delete 
an anchored stroke from the database or to acquire a list of all of the anchored strokes. 
40 [0049] Rgs. 4A through 4F show a ftow chart outlining a control routine for rnapping freeform digital 

toobjects in an underlying view in accordance with an embodiment of the invention. The flowchart parallels the fdbwing 
pseudocode. 
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First, classify the group ofstrokes as circles, underlines/cros&cuts, insertions* margin 
notes, or connectors. Notice that this may require splitting some of the strokes, and 
may result in several groups ofstrokes. Then, 

Sort the new groups ofstrokes so that connectors ^ear after any groups they 
connect to 

For each new group of Strokes 

If the group is a circle or an undertine or a cross-out, then 

Split the group of strokes into a list of strokes and diaracters 
For each ^ke and character pair in the list 

Create a new sin^>le anchor corresponding to the character 

Get the region corresponding to the character 

Let anchor position be the center of the region 

Create a new anchored stroke based on the single anchor 

For each portion in the stroke. 

Let offiet be position minus anchor position 
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Append o&set to the anchored stroke's ]ist of of&ets 
End for 

Add anchored stroke to the annotation database 
End for 

Else if the groi^ is an ink inseftion into the text» then 

Let character position be the charat^ position where the ink di|» into the 
text 

Get the position corresponding to the character position 
Let anchor position be that position 
For each stroke in the group 

Create a new anck^ stroke based on the character position 
For each position in the stroke 

Let ofTset be position nxinus anchor position 
Append offid to the anchored stroke's list of of&ets 
End for 

Add anchored stroke to the annotation database 
Endfor 

Else if the group is a margin annotation, then 

Let character position be die first character position corresponding to the 
vertical extent of the annotation 

Get the left and ri^ margin positions corresponding to character position 
If the group is mostly to die left of the left margin position, dien 
Let margin anchor be fdiaracter position, flag^lcft } 
Let anchor position be the left margin portion 

Else 

Let margin anchor be { character position, flag=rigl^} 
Let anchor position be die right margin position 

End if 

For each stroke in the group, 

Qeate a new anchored stroke based on the margin anchor 

For each position in the stroke 

Let offset be position minus anchor position 
Append offset to the anchored stroke's list of of&ets 

End for 

Add anchored stroke to the annotation database 
End for 

Else if the grotq) is a connector, then 

For each of starting and ending position on connector 
If position is sear an armotation 

Let simple anchor be the anchor for dial annotation 
Let anchor position be the current position of the simple 
anchor 

Let offset be position minus anchor position 
Else (position is near an object) 

Let simple ancl»ir bethe anchor fwthat c^ject 
, Let anchor position be the current positimi of the object 
Let of&et be position minus anchor position 

End if 
End for 
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Let composite anchor be the anchors and ofEsets coireq>onding to the 
starting and ending positions 
5 For each stroke in the group 

Create a new anchored stroke based on the composite anchor 
For each position in the stroke 

Let offset be position minus starting position 
10 Divide offset by the distance between tbestarting and 

ending positions 

Rotate of&et around the origin by-angle (ending position 
minus starting position) 
15 Append of&^ to the anchored stroke's list of ofi&ets 

End for 

Add andiored stroke to the anno tati on database 
End for 

20 Endif 
End for 

[0050] The ccntrol routine shown in Figs. 4A-4F starts at step SlOO and continues to step S102. in step S102» the 
control routine groups the ink strokes Into annotations such as circles, under1lnesA:ros&outs, insertbns, marginalia or 

^ connectors. It is to be appreciated that this may require splitting soma strokes and may result in several groups of 
strokes. The control routine then continues to step SI 04 where the annotations are sorted so that connectofs appear 
after any group to which they are connected. The control routine then continues to step SI 06 where the control routine 
initiates a kx)p that iteratfvely processes all d these arvtotations. The control routine determines it the current annotation 
is a circle, an underline or a cross-out. If h step SI 06, the control routine determines that the current annotation is a 

30 cirdo, an underline or a crossKHrt then the control routine jumps to step S1 16. If, on the other hand, in step SI 06 the 
control routine determines that the current annotation does not include a c^la, an underline or a cross-out then the 
control routine continues to step S108. in step S108, the control routine determines if the current annotation is an ink 
insertkin. if, in step SI 08, the control routine determines that the current annotation is an ink tnserton then the control 
routine jumps to step SI 38. It, on the other hand^ the control routine determines that the current annotatk)n is not an 

35 ink insartkm. then the control routine continues to step S110. In step S110. the control routine determines 9 the if the 
current annotatkyi is a margin annotatk)n. If» in step S110. the control routine detemDines that the current annotatbn 
is a margin annotatkx) then the control routine jumps to step SI 58. If, on the other hand, in step S 110, the control 
routine determines that the current annotatnn is not a margin armotatkm then the control routine jumps to step SI 86. 
[0051] After completing one of the processes of Figs. 4&4F, which wai be explained in detail t»ek>w, the control 

40 routine continues to step S1 12 where the control routine determines if more annotattoris need to t>e processed. If in 
step S11 2. the control routine detemitnes that more annotatkxis need to be processed, then the control routine returns 
to step S106. If, on the other hand, the control routine determines that no more annotations need to be processed, 
then the control routine continues to step S1 1 4 where control is returned to the environment in v^ch the control routine 
of Figs. 4A-4F operate. 

^ [0052] Al step S1 1 6, the control routine splits the annotatkm intoa list of substrokas each with a corresponcfing single 
character. Thus, the control routine splits the annotation into substroke^character pairs. The control routine then corv 
tinues to step Slid, where the control routine inftiates a kxp to iterativety process each sutxstroke/charader pair in 
the current annotatkm. tn step S1t8, the control routine creates a new s&hple anchor that corresponds to the current 
character. The control routine then contmuss to step Si 20 where the control routine acquires the regkm in the view 

so that corresponds to the current character The control routine then continues to step SI 22 where the control routine 
sets an anchor positkm to the center ol the tegon m the vtawi The control routine then contmues to step S124 where 
the control routine creates a new anchored stroke that is based upon the simple anchor and continues to step SI 26. 
In step SI 26, the control routine sets an offset to the position from the substroke minus the anchor positnn and con- 
tinues to step SI 26. In step SI 28, the control routine appends the offset to the anchored stroke's list ol offsets and 

S5 continues to step SI 30. In step SI 30, the control routine determines whether more positcns exists in the current 
substroke. If the control routine dstermbies in step SI 30 that more positkxis exists in the current substroke, then the 
control routine returns to step SI 26. If. on the other handl the control rout^e delermmes that more positbns do not 
exist in the current substroke then the control routine continues to step SI 32. In step SI 32. the control routine adds 
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the anchored stroke to the annotation datat>ase, and the controt routine then continues to step SI 34. In step SI 34, 
the control routine determines if more suttstroka^character pairs exist tn the current annotation, II, in step SI 34, the 
control routine detemnines that more substroke/icharacter pairs exist on the current annotalkan then the control routine 
returns to step S118. It. on the other hand, the control routine determines, in step S134. that no more substroke^ 
5 character pairs exist in the current annolatwn then the control routine continues to step Si 36 where control returns to 
stepS11^ 

[0053] AtstepSISBthecontrdroutinesetsacurrentcharacterpositiontothecharacterpositkxiM^^ 

dips into the text. The control routine then continues to step S140 where the control routine acquires the positkxi 

corresponding to the oirrent character positkyi. The control routine then continues to step S142 where an andKV 

10 po6itk)n is set to the current character position. The control routine then continues to step SI 44 where the control 
routine initiates a kiop to process aQ strokes in the annotatksa In step SI 44. the control routine creates a new anchored 
stroke t>a8ed upon the current character position as an anchor and the control routine continues to step SI 46. tn step 
SI 46. the control routine sets an offset to the current character positkm from the stroke minus the anchor positksn and 
continues to step SI 48. In step S148. the control routine appends the oftset to the anchored stroke's list of oftsets and 

IS continues to step SI 50. In step SI 50. the control routine detemttnes whether more posrtkxis exists in the current stroke. 
If, in step SI 50. the control routine determines that more positkans exist in the current stroke tfien the control routine 
returns to step SI 46. If, on the other hand, in step S150. the controt routine detemiines that no more posdkns exist 
in the current stroke then the control routine continues to step SI 52. In step SI 52. the control routine adds the anchored 
stroke to the annexion database and the control routine continues to step S154. tn step S154. the control routine 

20 determines whether more strokes exist m the current annotatfon. II. in step SI 54. the control routine determines that 
more strokes exist in the current annotatkm then the control routine returns to step S144. If, en the other hand, the 
control routine determines, in step S154. that no more strokes exists in the current annotation then the control routine 
continues to step S112. 

[0054] At step S158, the control routine sets a character positnn to the first character positbn corresponding to the 
^ vertcal extent of the current annt^tioa The control routine then continues to step SI 60 where the text editor acquires 

the left and ri^ margin positbns that correspond to the character positkyi and continues to step S16^ In step S162. 

the control routine determines whether the annotation is mostly to the left of the left margin. If, on step SI 62, the control 

routine detemunes that the annotation is mostly to the left of the left margin, then the control routine continues to step 

Si 64. K. on the other harxJ, the control routine detennines that the annotatkm is not mostly left of the left margin then 
30 the control routine jumps to step S16B. In step SI 64, the control routine sets the margin anchor to LHFT and the control 

routine continues to step S1 66. In step SI 66. the controt routffie sets an anchor positnn to the left margm position and 

continues to step Si 72. 

[00S5J In step SI 68. the control routine sets a margin anchor to RIGHT and continues to step SI 70. In step SI 70, 
the control routine sets an anchor positk)n to the right margin position and continues to step S172. In step S172. the 

35 control routine initiates a fcwp to process all strokes in the current annotation. In step S172, the control routine creates 
a new anchored stroke based upon the margin anchor and continues to step SI 74. In step SI 74. the control routbte 
sets an offset to the position from the current stroke mmus the anchor position and continues to step SI 76. In step 
S176. the control routine appends the offset to the anchored stroke's list of offsets and continues to step 8178. In step 
S178. the control routine determines whether more po^tkyis exist in the current mroke. If. m ^ep S178, the control 

^ routffte determines that more positions exist in the current stroke then the control routine returns to steps S174. It, on 
the other hand, In step S176, the control routine determines that no more positions exists in the current stroke then 
the control routine continues to steps SI 80. In step SI 80. the control routine adds the anchored stroke to the annotation 
database and continues to step SI 82. tn step SI 82. the control routine detennines whether more strokes exists in the 
current annotatbn. If, in step SI 82. the control routine detennines that more strokes exists in the current annotatksn. 

^ then the control routine r^ums to step S172. If, on the other hand, the control routine determines, in step 8182. that 
no more strokes ext^s in the current annotation then the control routine continues to step SI 56 where the control 
routine returns to step S11 2. 

[tX3S$l At step S186, the control routine initiates a kx)p to process all starting or ending positions in the cunent 
annotation. In step S186, the control routine determines whether the current ol the starting and ending posKkms are 

^ near another annotatnn or an object If in step SI 86. the control routine determines that the current position is near 
another annotation then the controller continues to step SI 88. If, on the other hand, the control routine detennines, in 
step S166. that the position Is not near another annotation then the control routine continues to step S192. tn step 
S188, the control routine sets a simpts anchor as the anchor for the current annotatkxi and continues to step SI 90. In 
step S 190. the control routine sets an anchor positkxt as the current posttkm of the simple anchor and continues to 

« step SI 96. In step S192, the control routine lets a simple anchor be the anchor for an object which is near the current 
starting or endmg positkin and the control routine continues to step SI 94. In step SI 94. the control routine sets an 
anchor position to the current positbn of the object and the controt routine continues to step S196. In step S196. the 
control routine sets an offset to the current posilcn minus the anchor position and continues to step SI 98. In step 
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S198, the control routine determines whether more starting or ending positions exist in the current annexion. If. in 
step S1S8. the control rcKJtine determines that more starting or ending posSions exist in the current annotation then 
the control routine returns to step S186. If, on the other hand, the control routine detennines, in step S198, that no 
more starting or ending positions exist in the current annotation then the control routine continues to step S200. 

5 [00S7] In step S200, the control routine initiates a loop to process all strokes in the current annotation. In step S200. 
the control routine aeates an anchored stroke having a composite mhor based upon the anchors corresponding to 
the starting and ending posittor^ and continues to step S202. In step 8202, the control routine sets an offset to the 
current positkxi in the current stroke minus the starting posdion and continues to st^ S204. In step S204, the control 
routine di\ndes the offset bfy the distance between the starting and ending positions and continues to step S208. In 

<o step S206. the control routine rotates the offset around an origin by an angte equal to the ending posdion minus the 
starting position and continues to step S208. In step S208, the control routine appends the offset to the anchored 
stroke*s list of offsets and continues to step S210. In step S210, the control routine determines whether nrx)re positions 
exists in the current stroke, tl. in step S210. the control routine dstemnines that more posftions exists in the current 
stroke then the control romine returns to step S202. It* on the (^er hand, the control routine determines, in step S210, 

IS that no more positions ex^ in the current stroke then the control routine continues to step S212. In step S212, the 
control routine adds the compose anchor to the annotation database and continues to step S214. In step S214, the 
control routine determines if more strokes exists in the current annotation. If, in step S21 4, the control routine detennines 
that more strokes exists in the current annotation then the control routine returns to step S200. If. cn the other hand, 
the control routine determines that no more strokes exists tn the current annotation, then the control routine continues 

20 to step S21 6 where the control routine returns to step S11 2. 

[0058] Figs. 5A-5B shows a control routine for rerxtoring freeform cfi^ hk annotations on a changed vtewi The 
flow chart parallels the followtng pseudocode. 

2s Using the aniK>tadon database, get the list ofaO anchored strokes 

For each anchored stroke* 

Extract the anchor and the list of oi&ets 
Let stroke be a new stroke 
^ If anchor is a simple anchor 

Detennine if the character or character position underlying the anchor 
has been deleted 
If not, then 

Let anchor portion be (he position of that anchor 

35 
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For each oflset. 

Let new position be oflset + anchor position 
Add new position to the stroke 
End for 

End if 

Else 

Detennine if the character or character positions underlying the 1" and 
2°^ anchors has been deleted 
If both have been deleted, then 
do nothing 

Else 

Let starting position be the position of the l** anchor, plus the 
l-'oflfeet 

Let ending position be the position of the 2"* anchor, plus die 
2"^ oflset 
For eadi offset, 

Let new offs^ be offset rotated byangle (ending 
position minus starting position) 
Multiply new ofi&et by the distance between the starting 
and ending positions 

Ld new position be new offset plus starting position 
Add new position to the stroke 
End for 

End if 

End if 

Render stroke on the view 
End for 

3S [0059] Thocontrdioutjnestansatst6pS300arxJcontinua3tost6pS3f>2.ln3tepS30^ 

a list of aD anchored strokes from the annotation database and continues to step S304. In step S304, the control routine 
extracts the anchors and the list ot offsets from the current anchored stroke and continues to step S306. In step S306, 
the control routine lets the stroke be a new stroke and continues to step $308. In step S308. the control routine deter- 
mines whether the anchor is a simple anchor. II. tn step S308. the control routine determines that the anchor is a simple 

40 anchor then the control routma continues to step S310. tl, on the o^er hand, the control routine determines, in step 
S308. that the anchor is not a simple anchor then the control routine jumps to step S326. In step S310, the control 
routine detennines if the character or character positksn has been deleted, tf. in step S310, the control routine deter- 
mines that the character or character positkxi has been deleted then the control routine jumps to step S320. II. on the 
other hand, the control routine determines that the character or the character position has not been deleted then the 

^ control routine then contonues to step S31 2. In step $31 2, the control rou&ie sets the anchor position to Uie position 
of that anchor tor the off s^ and continues to step S31 4. In step S314, the control routine lets the new position be the 
offset Irom the stroke plus the anchor position and continues to step S316. In step S316. the control routine adds the 
new pos^kjn to the stroke and continues to step S3t8. In step S318. the control routine detennines whether there are 
more offsets for the anchor. II. in step S318. the control routine determines that there are more offsets then the control 

so routine returns to step S314. II. on the other hand, the control routine detemrtines that there am no more onsets then 
the control routine jumps to step S320. In step S320. the control routine renders the stroke on the view and oont'nues 
tostep $322. In step $322. the control routine ddternrunes if them are more anchored strokes in 
II. in step S322. Uie control routine determines that there are more anchored strokes in the annotatk)n database then 
the control routine returns to step S304. If. on the other hand, the control routine, delermones in step 8322. tttat there 

ss are no more anchored strokes in Uia annotatnn database then the control routine continues to step S324 where Uie 
control routine returns control to the environment in whk^ Uie control routine operates. 

[00601 In step S326. the control routine detemriines if the character or character position underlying Uie first and 
second anchors been deleted. II. In step S326, Uie control routine determines that both d these character or character 
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positions have been deleted then the control routine returns to step S320. If. on the other hand, the control routine 
deterrmnes. tn step S326. that the charader or character position undertintng the first and second anchors have not 
been deleted then the control routine continues to step S328. In step S328, ^e control routine sets the starting position 
to the position of the first anchor plus the first offset and continues to step S330. tn step S330. the control routine sets 

s the ending position to the position of the second anchor plus the second offset and continues to step S332. In step 
S^, the control routine sets the new offset b the offset from the anchored stroke rotated t»y an angle equal to ending 
position nuinus starting position and the control routm contmues to step S334. In step S334, the conUol routine mu)- 
tipBes the new offset by the distance between the starting and encfing positions and continues to step S336. In step 
S338, the control routine sets the new position to the new offset ptus the starting position and continues to step S338. 

10 In step S338, the control routine adds the new position to the stroke and continues to step S340. In st^ S340. the 
control routine determines whether mc^e offsets exist If. in step S340. the control routine determsies that more c^sets 
ex^ then the control routine returns to step S332. If. on the other hand, the control routine determines, in step S340. 
that no more offsets exists then the control routma returns to step S320. 

[0061] Freefomi digital ink annotation is intended to rnean a visible ink stroke of arbtt a 
IS mouse or other pointing device that persists over time. A pen based input device is ontended to include a pen tablet, 
a pen display, a pen display tablet, a pen computer andtor a touch screen. The systems and methods of this inv^ion 
may infer some meaning from the shapes or the morphotogical characlerfetfcs of ink strokes but do not renrwve the ink 
stroke once they have been recognized. 

[0062] White the detailed description refers genemlly to electronic systems, it is intended to be understood that the 
so method and system of the preswit invention may be applied to any electronic book and document reader that displays 
changing views. Electronk: book and document readers indude any electronc handheld devce that can be accessed 
with a pointing devk:6 such as a pen and that accepts freefomn digital ink as input Electrons book and document 
readers may also include other additional features without Mtatkxi. 

[0063] ft is tntencted to be understood that the term texT is intended to include text, dgital ink. audio, video or any 
2S other content of a docurrtent. hctudtng the document's strudure. An object is intended to be any component of a display 
such as. by way of non-ltmding example, a character, a word, a phrase, a sentence, a paragraph, a line, a sectkin, a 
cfiapter. a portnn of an Image that ^ identifiable by a computer vision system m a single image or in a stream of images 
as tn a vkteo, etc 

[0064] It sftoutd be understood tfrnt a change h the structure of a document is intertded to include inserting, deleting. 
^ copying and moving of objects or the like. It is also intended to be understood that a change in the layout stnicture of 
a document includes r^lowing of objects, restzffig of objects, rotation of objects, stretcheag of objects, distortion of 
objects, morphing of objects or the like. 

[0065] While this Invention has been described in conjunctkin with the specific embodin^nts outlined above, it is 
evktent ttiat rruiny alternatives, modificatrorts arxl variatbns are apparent to those skilled in the art. Aocordlngty, the 
^ preferred embodiments of the inventkm as set fodh above are intended to be Qtustrated and not iimiting. Vbrk>us chang- 
es may be made without departing from tf>e spirit and scope of the invention. 



Ctalma 

40 

1. A method for nnaintahing an tndcatnn of a logical retatKmsh^ of at least one freeform digital ink annotatkxi to at 
least one object in a first view as the first view changes, the indication inctudtng a layout relationship ot the at least 
one freeform digital ink annotation to the at least one object, the method comprising: 

^ mapping the togcal retatbnship of the at feast one freeform digital ink annotation to the at least one object 

based upon the layout retatkxishtp of the at least one freeform digital ink annotation and the at least one ot^ect 
in the first view, 
generating a second view; 

rendering the at least one freeform digital ink annotatkxi in response to the generation of the second viem( the 
50 layout relationship of the at least one f reef onm digital ink annotation and the at least one object in the second 

view based upon the mapped logical relationship of the at least one freeform digital ink annotatkxi to the al 
least one object in the first view. 

2. TTie ntethod of daim 1 . wherein the second view is generated in response toat least one of a change in the stnicture 
^ of a document and a change in the layout of the document 

3. TTie method of claims 1 or 2. wherein mapping the togical relationship comprises: 
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splining the at least one freeform digital ink annotation into a ptura&ty oi sutxstrokes, each substroka oone- 
spcndngtoaportionof theannotalton thatistnproxiniity to an object; and 

creating an anchor (or each of the plurality of siitt^rokes based upon the layout retatbnship of each of the 
plurality of substrokes to at least one object rn the first view 

5 

4. The method of daim 3, wherein: 

rendering the at least one freeform digftal ink annotation comprises rendering the at least one f reefonn digital 
ink annolatnn in response to a separation of the plurality of objeds in the second view; and 
10 splitting the at leasi one freeform digital ink annotatk)n further comprises splitting the at least one free form 

digital ink annotation cnto at teasi tm freeform digha) ink armotatkxis, each freeform digital Ink annotation 
corresponding to at least one of the plurality of objects. 

5. The method of any of claims 1 -4, wherein: 

15 

the at least one object comprises at least two objects; and 

gerierating the second view comprises generating the second view in response to at least one of merging at 
least two of the at least two objects and replacing at least one object of the at least two objects with at least 
one new object 

so . 

& The method of any of claims 1-5, further conr^)ftsing mapping the k)gicat relattonsh^ based on at least one of a 
shape of the at least one freeform d^l ink annotatkm, a proximity of the at least one freeform digital ink annotation 
to the object, and a bcatkyi of the at least one freeform digital bik annotatkm in the first view 

2S 7. A system for maintaining an tmfcation of a logical relatkxishp of at least one freefomi digital ink annotation to at 
least one object in a view as the view changes, the indicatkx) including a layout retattonshp of the at least one 
freeform digital Ink annotatkxi to the at least one object, the system comprising: 

a display that displays the layout retatkir^ip of the at least one freeform digital ink annotation to the at least 

30 one object; and 

a processor that maps the at least one freeform digital ink annotation to the at least one object based upon 
the layout relationship of the at least one freefonn digital ink annotatkxi to the at least one object in a first viev< 
the (^lay [being] responsive to a change tn the first view to generate a second view and to render the at 
least one freeform digital ink annotation in the second view, the layout relationship of the at least one freeform 

^ digital ink annotatkm and the at least one object in the second view based on the mapped bgkal relationship 

of the at least one freeform digital ink annotatran to the at least one object tn the first view. 

a The system of daim 7, wherein the procassomnaps tf^ at least one freeform digital ink annotatkin by splitting the 
at least one freeform digital ink annotation into a pluralrty of substrokes. each substroka corresponding to the 
^ portk)n of the freeform digital ink annotatkin that Is in proxtmity to an object, and by creating an anchor for each 
of the pluraKty of substrokes based upon the layout relatkytsh^ d each of the plurality of substrokes to at least 
one object in the first viewi 

9. The system of clamts 7 or 8. wherein the processor generates the second view in response to at least one of the 
^ at least one object being split into at least two objeds and the at least two objeds being separated, the processor 

splitting the at least one freeform (figital ink annotation Into at least two freeform digital ink annotatbns. each 
freeform digital ink annotaikn corresponding to at least one of the at least two objects. 

10. The system of any of claims 7-9, wherein: 

so 

the at least one objed comprises at least two objeds; and 

the processor generates the second view in response to at least one of at least two of the at least two objects 
being merged together and at least one of the at least two objects being replaced with at least one new object 

ss 11. The system of any of claims 7-10. wherein the processor maps iha at least one (reetomt digital ink annotatnn 
based on at least one d a shape ^ the at least one freeform cfigital ink annotatkn a proxt^ 
freeform digital ink annotalk)n to the at lec^ one objed and a kxatnn of the at least one freefomt di^ ink 
annotatbn in the first view. 
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12. Thesystemdany of claims 7-11. wherein: 

the at one freetoim digital nkamMtation includes at least two Ireeforrn digital ink annotations and tho at 
least one object includes al least two objects, each object corresponding to one ot the at least two f reeform 
s digital ink annotatxyis; and 

the processor merges at least two ot the at least two fresform digital ink annotations into a singto freetorm 
digrtal ink annotatkm in response to at least two of the at least two objects being nnerged. 
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